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Software  Engineering  Institute 

•Established  in  1984  -  Applied  R&D  Laboratory  situated  as  a 
college-  level  unit  at  Carnegie  Mellon  University 

•DoD  staff  ceiling  (FY04):  147;  Technical  staff  of  315 


Offices  in  Arlington,  Va,  Pittsburgh,  Pa,  Red  Stone  Arsenal, 
Al,  Colorado  Springs,  Co,  Frankfurt,  GE 


Mission:  Provide  the  technical  leadership 
to  improve  the  practice  of  Software 
Engineering  so  the  DoD  can  acquire  and 
sustain  its  Software  Intensive  Systems 
with  predictable  and  improved  Cost, 
Schedule,  and  Quality 


Goal:  Institutionalize  new  and 
improved  practices  in  the  acquirer 
and  developer  communities 
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SEI  Strategy 

Helping  others  make  measured 
Improvements  in  their 
software  engineering  practices 
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SEI  Research  Agenda  -  Create 

_  The  right  software  delivered  _ 

defect  free,  on  cost,  on  time,  every  time 
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product  lines 


Integration  of 

Survivable 

SIS 

Systems 

Performance 

Product 

Critical  Systems 

Line  Practice 

Software 

Predictable 

Architecture 

Assembly 

Technologies 

with 

Certifiable 

Components 

Technical  Practice 
Initiatives 


with  predictable  and  improved 
cost,  schedule,  and  quality 

Capability  Team 

Maturity  Software 

Model  Process 

Integration 

Software 
Engineering 
Measurement 
&  Analysis 


Management  Practice 
Initiatives 


©2003  by  Carnegie  Mellon  University 


page  6 


Carnegie  Mellon 

Software  Engineering  Institute 


Apply 


Lessons  and  Practices 
Transitioned  Widely 
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What’s  Important? 


Carnegie  Mellon 

Software  Engineering  Institute 

Defining  the  Processes 
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Improving  the  Processes 


Treating  the  cause 
rather  than  the 
symptoms. 


Acceptable 

product 
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CMMI  in  a  Nutshell 

CMMI  provides  guidance  for  improving  an  organization's 
processes  and  ability  to  manage  the  development, 
acquisition,  and  maintenance  of  products  or  product 
components. 

CMMI  places  proven  approaches  into  a  structure  that 

-  helps  your  organization  examine  the  effectiveness 
of  your  processes 

-  establishes  priorities  for  improvement 

-  helps  you  implement  these  improvements 

Improving  processes  for  better  products 
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Why  Focus  on  Product  Development? 

A  system’s  engineering  approach  is  critical  for  today’s 
extremely  complex  DoD  systems. 

•  Essential  for  successful  Spiral  Development  and 
(Evolutionary)  Acquisition  process 

•  Critical  for  successful  Technology  Insertion  and  Technology 
Transition  for  modern  systems 

Recent  example:  Lack  of  robust  systems  engineering 
practices  identified  as  critical  factor  in  SBIRS-High  problems 

(per  Lt.  Gen.  Brian  A.  Arnold,  USAF,  CDR,  USAF/SMC,  5/6/02  Aviation  Week) 


CMMI  implementation  is  major  forcing  function  for  the  needed 
systems  engineering  content  of  today’s  systems 
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Complexity  in  Modern  Systems 

Many  commercial  products  are  the  result  of  a  complex  mix  of 
subcomponents  engineered  into  a  system 


Most  DoD  weapon  and  information  systems  are  at  least  this 
complex 


Active  Roll  Statnlizalion  1  APS)  optional  rear- seat  aide-impact 
Front-seat  Head  O  airbags  [deactivation  option! 

Protection  Syslem  |HPSj  O  with  TeBt  Head  Protection  System 

and  aide-impact  aiitoaga  Q  i  Drl  «a  concept  ° 

a.e-htw.  e-5oeed  adenUve  Acdw#  Knee  Protection 

ass-hofaepaarar  O  O  a.tomalic  tra-ssmiaalon  O  aid  Head 

¥-8  engine  Restraints  for  Iront  seats 

ihls^’wamheM^ntTwith'  O  Optond  Sport  Package 

dynamic  auto-leveling  O  ftont  36313  O 

Messlve  ABB  ventilated  Dynamic  StatnEity  Oontrol  1USD) 
disc  brakes  with  with  All  Season  Traction  (AsC)  and 

Dynamic  Brake  Control  Dynamic  Traction  Control  (DTC! 
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Weapon  System  Complexity 
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Increasing  System  Complexity 
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Focus  of  CMMI 


CMMI  is  applied  here 


SW-CMM  is  applied  here 
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CMMI  SE/SW/IPPD/SS 


•  Organizational  Process  Focus 

•  Organizational  Process 
Definition 

•  Organizational  Training 

•  Organizational  Process 
Performance 

•  Organizational  Innovation 
and  Deployment 


•  Project  Planning 

•  Project  Monitoring  and 
Control 

•  Supplier  Agreement  Mgmt. 

•  Integrated  Project  Mgmt. 

•  Integrated  Supplier  Management 

•  Risk  Management 

•  Quantitative  Project  Mgmt. 

•  Integrated  Teaming 


•  Requirements  Management 

•  Requirements  Development 

•  Technical  Solution 

•  Product  Integration 

•  Verification 

•  Validation 


•  Configuration  Mgmt. 

•  Process  and  Product 

•  Quality  Assurance 

•  Measurement  & 
Analysis 

•  Decision  Analysis  and 
Resolution 

•  Causal  Analysis  and 
Resolution 

•  Organizational 
Environment  for 
Integration 
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CMMI  Steering  Group 


Bob  Rassa,  Raytheon  (Co-chair) 
Mike  Nicol,  USAF  (Co-chair) 

Ric  Sylvester,  OUSD(AT&L) 
Dave  Castellano,  OUSD(AT&L) 
George  Desiderio,  OUSD(AT&L) 
Brenda  Zettervall,  USN 
vacant,  USA 


Clyde  Chittister,  SEI 
Bill  Peterson,  SEI 

Hal  Wilson,  Northrop  Grumman 
Bob  Lentz,  General  Dynamics 
Joan  Weszka,  Lockheed  Martin 
Leroy  Brown,  Motorola 
Linda  Ibrahim,  FAA 
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Agenda 

SEI  Overview 
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Acquisition  use  of  CMMI 

Acquisition  organizations  can  use  the  CMMI  to: 

•  Help  discriminate  between  offerors  during  a  competitive 
source  selection 

•  Help  incentivize  contractors  to  use  effective  practices 
and  improve  those  practices  after  contract  award 

•  Establish  an  acquisition  process  improvement  program 
within  the  program  office 
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Critical  Questions  for  Source 
Selection 

If  I  require  everyone  to  be  Maturity  Level  3,  is  it  a 
discriminator  or  a  non-discriminator? 

Is  process  maturity  of  the  development  teams  important 
enough  to  be  a  discriminator,  can  I  really  find  out  without 
checking  the  behavior  of  the  organization? 

If  it  is  important  enough,  do  I  have  the  time  and  resources 
to  check? 
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What  is  Maturity  Level  3? 

If  an  organization  is  a  Maturity  Level  3  developer,  you  can 
expect  on  their  next  project 

-  Team  experience  as  captured  in  processes  is  based 
on  organizational  guidance 

-  Estimates  are  based  on  historical  data 

-  The  organization  continually  assesses  their 
processes  and  products  to  look  for  improvement 

-  Training  is  defined  and  provided 

-  Stakeholders  are  involved 

-  Engineering,  Management,  Support,  and  Process 
related  practices  are  defined,  used,  measured,  and 
improved 
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Real  Life 


CMMI  Math:  3  +  4  +  5  +  1=  ? 
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Implications 

Maturity  Levels  are  a  good  starting  point 

Need  to  ensure  the  team’s  practices  are  sound  and  that 
risks  associated  with  the  way  the  team  does  business  are 
continually  identified  and  addressed 

The  acquisition  team’s  practices  impact  the  team’s  overall 
performance 
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Contract  Monitoring  Example 

National  Reconnaissance  Office 

Freedom’s  Sentinel  in  Space:  One  Team, 
Revolutionizing  Global  Reconnaissance 

Mission  of  the  NRO:  Enable  U.S.  global  information 
superiority,  during  peace  through  war.  The  NRO  is 
responsible  for  the  unique  and  innovative  technology,  large- 
scale  systems  engineering,  development  and  acquisition, 
and  operation  of  space  reconnaissance  systems  and  related 
intelligence  activities  needed  to  support  global  information 
superiority. 
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System  Characteristics 

Huge  system  engineering  endeavors  encompassing  space 
vehicles  and  ground  infrastructure 

Complex  software  engineering  and  hardware 
responsibilities 

System  development  pose  big  risks  in  acquisition 
programs 

•  Several  Million  SLOC  programs 

•  Dispersed  engineering  &  development  locations 

•  Multi-contractor  teams  using  different  processes 

•  Combination  of  legacy  re-use,  COTS  integration 
and  new  software  development  efforts 

•  Real  cost  and  schedule  constraints 
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Strategic  Plan  for  Insight 

Year  0  Year  2  Year  3  Year  4  Year  5 

-A  — AfA - 1 - A— Al— A— A— I-  -  - 


Year  0: 
Year  2/3  : 
Year  4: 
Year  5  .... 


Conduct  series  of  source  selection  appraisals  for  all  Offerors 
Conduct  baselining  appraisals  for  primes  and  subcontractors 
Conduct  “delta”  appraisals  for  primes  and  subcontractors 
Conduct  “statusing”  appraisals  for  primes  and  subcontractors 


Source  Selection  Appraisals 


Baselining  Appraisals 
Delta  SCE  /  Statusing  Appraisal 
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Results  of  Contract  Monitoring 
Appraisals 

Findings  from  all  sites  combined  into  a  set  of  “program  findings” 

•  t  684  Program  Findings  (specific  problems  or  strengths) 

•  (~  55%  program  strengths;  ~  45%  weaknesses  =>  risk 
areas) 

“Affinity  Grouped”  Weaknesses  to  correct  systemic  problems,  not 

just  symptoms 

•  For  example:  “Baseline”  Management  would  combine 
findings  from  CM,  RM,  RD,  TS,  etc. 

1 1  Risk  areas  /  Process  Improvement  Categories  identified 

•  Being  used  as  the  basis  for  project  process  improvement 
activities 
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Weakness  Characterization 
by  Process  Grouping 
Across  Program 
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Project  Mgmt  Engineering 


■  Weaknesses/Improvement  Activities 


Project  Mgmt  Processes: 

-  Project  Planning 

-  Project  Monitoring  &  Control 

-  Integrated  Project  Mgmt 

-  Risk  Management 

Engineering  Processes 

-  Requirements  Mgmt 

-  Requirements  Definition 

-  Technical  Solution 

-  Product  Integration 

-  Verification  (Peer  Reviews) 

Sunnort  Processes 

-  Measurement  &  Analysis 

-  Product  and  Process  Q  uality 
Assurance 

-  Configuration  Mgmt 

-  Decision  Analysis 

Process  Mgmt 

-  Organizational  Process  Focus 

-  Organizational  Process  Definit 
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Issues  Identified  in  Appraisals  - 
Program  Management 

Use  of  corporate  standard  engineering  processes  on  program 
Lack  of  project  plans  or  having  only  incomplete,  conflicting  or  out  of 
date  project  plans 

Ineffective  use  of  Integrated  Master  Schedule  as  basis  for 
planning/tracking  status  across  program 
Undefined  engineering  and  management  processes  on  program 
Inability  to  track  and  manage  actions  to  closure 
Cost  estimation  processes,  methods,  data  and  tools 
Staffing  and  training  project  personnel 
Tracking  dependencies  between  or  across  teams 
Managing  project  data 

Ability  to  proactively  identify  and  manage  risks 


©2003  by  Carnegie  Mellon  University 


page  32 


Carnegie  Mellon 

Software  Engineering  Institute 


Issues  Identified  in  Appraisals  - 
Engineering 

Understanding  of  the  program’s  requirements 
Requirements  traceability  to  architecture/design  or  to  test 
plans/procedures 

Linkage  of  functional  and  performance  requirements 
Inconsistent  requirements  management  at  different  levels 
Criteria  for  making  architectural/design  decisions  among 
alternatives 

Capturing  entire  technical  data  package  (requirements,  design 
and  design  rationale,  test  results,  etc) 

Efficiency  of  design  process/methods 
Defining  integration  and  test  procedures 
Defining/maintaining  integration  and  test  environments 
Existence  of  integration  procedures 
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Issues  Identified  in  Appraisals  - 
Support  Processes 

Identifying  items  in  configuration  management  baselines 
Ability  to  manage  individual  “versions”  in  incremental  development 
Effectively  managing  changes  to  work  products  throughout  lifecycle 
Conducting  audits  to  establish/ensure  integrity  of  baselines  throughout 
incremental  engineering  and  development 
Effectiveness/efficiency  of  change  management  process  (cycle  time, 
volume  of  changes) 

Roles/responsibilities  of  change  control  boards 
Quality  Assurance  audits  of  products  and  processes 
QA  involvement  in  system  and  software  engineering  processes 
Sufficiency  of  resources  for  quality  assurance/product  assurance 
Defining,  storing,  analyzing,  using  measurement  data 
Breadth  of  metrics  to  manage  engineering  activities  (outside  of 
cost/schedule  data) 
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Progress  In  Action-Plan 
Implementation 

Re-Assessed  during  subsequent  appraisals  (18  months  later) 

•  Good  News:  Majority  of  issues  addressed  or  completely  resolved 

•  One  program  segment  (prime  and  subcontractor  teams) 

-  73  findings  resulted  in  41  Action  Plans  through  affinity 
grouping 

-  Thirty  (30)  were  implemented  within  6  months  of  appraisal 

-  Additional  eight  (8)  implemented  within  9  months  of  appraisal 

-  Final  3  resolved  prior  to  return  appraisal 

-  Program  Mgmt  (contractor  and  gov’t)  briefed  weekly  on 
progress 

-  Contractors  gather  “evidence”  of  process  use  and 
effectiveness 

•  Major  Subcontractor: 

-  31  Findings  resulted  in  24  action  plans 

-  24  corrected  within  9  months  of  appraisal 

•  Additional  Subcontractor: 

-  22  findings  resulted  in  22  action  items 

-  All  22  corrected  within  6  months  of  appraisal 
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Bottom  Line 

In-progress  reviews  ensure  the  practices  used  by  the 
entire  team  are  effective 

Early  identification  and  mitigation  of  common  process- 
related  issues  and  problems 
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Acquirer/Supplier  Mismatch 
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Some  Acquisition  Scenarios 

Scenario  1:  Acquiring  a  low-risk  sub-component 
Scenario  2:  Acquiring  subsystems 
Scenario  3:  Acquiring  whole  systems 
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Scenario  1  -  Low  Risk  Component 

Project  X  is  building  a  mission  planning  system  to  manage  the 
tasking  of  an  earth  observing  sensor. 

Subsystems  include: 

Scheduling  Subsystem 
Planning  Subsystem 
Task  Management  Subsystem 
Reporting  Subsystem 
Map  Subsystem 

The  project  has  decided  to  procure  a  commercially  available 
mapping  system  for  their  map  subsystem.  Multiple  suppliers 
have  adequate  products  that  require  minimum  modifications  for 
the  purpose.  The  acquisition  team  would  need  to  help  analyze 
options,  select  a  supplier,  and  manage  the  supplier  agreement. 
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CMMI  SE/SW 


•  Organizational  Process  Focus 

•  Organizational  Process 
Definition 

•  Organizational  Training 

•  Organizational  Process 
Performance 

•  Organizational  Innovation 
and  Deployment 


•  Project  Planning 

•  Project  Monitoring  and 
Control 

•  Supplier  Agreement  Mgmt. 

•  Integrated  Project  Mgmt. 

•  Risk  Management 

•  Quantitative  Project  Mgmt. 


•  Requirements  Management 

•  Requirements  Development 

•  Technical  Solution 

•  Product  Integration 

•  Verification 

•  Validation 


Support 


•  Configuration  Mgmt. 

•  Process  and  Product 

•  Quality  Assurance 

•  Measurement  & 
Analysis 

•  Decision  Analysis  and 
Resolution 

•  Causal  Analysis  and 
Resolution 
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Scenario  2  -  Shared  Risk 

Project  Y  is  responsible  for  delivering  an  integrated  ground 
system  for  a  new  earth  observing  sensor. 

Ground  System 

Project  Y 


Mission  Planning 

Mission  Management 

Command  &  Control 

Systems  Control 

Training 

Subsystem 

Subsystem 

Subsystem 

Subsystem 

Subsystem 

Project  Y 

Subcontractor  A 

Sensor  Contractor 

Project  Y 

“Sister”  Division 

GFE 

Within  organization 

Success  of  Project  Y  is  highly  dependent  on  success 
of  suppliers  -  risk  of  failure  is  high  if  any  one  of  the 
suppliers  fail  -  the  project  needs  to  proactively 
manage  the  supplier  relationships. 
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CMMI  SE/SW/IPPD/SS 


•  Organizational  Process  Focus 

•  Organizational  Process 
Definition 

•  Organizational  Training 

•  Organizational  Process 
Performance 

•  Organizational  Innovation 
and  Deployment 


•  Project  Planning 

•  Project  Monitoring  and 
Control 

•  Supplier  Agreement  Mgmt. 

•  Integrated  Project  Mgmt  w/IPPD 

•  Integrated  Supplier  Management 

•  Risk  Management 

•  Quantitative  Project  Mgmt. 

•  Integrated  Teaming 


•  Requirements  Management 

•  Requirements  Development 

•  Technical  Solution 

•  Product  Integration 

•  Verification 

•  Validation 


•  Configuration  Mgmt. 

•  Process  and  Product 

•  Quality  Assurance 

•  Measurement  & 
Analysis 

•  Decision  Analysis  and 
Resolution 

•  Causal  Analysis  and 
Resolution 

•  Organizational 
Environment  for 
Integration 
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Scenario  3  -  Acquiring  Whole  Systems 

Project  Z  is  an  acquisition  organization  responsible  for  acquiring 
an  integrated  ground  system  for  a  new  earth  observing  sensor. 


-  Requirements 

-  Plans 


Integrated 

-  Acquisition 

Integrated 
Ground  System 

Ground  System 
Program 

cz)  Strategy  cz) 

-  Leadership 

Government  and  Insight 

Program  office  z  .  Solicitations 


-  Task  Orders 

-  Awards 

-  etc. 

Success  of  the  government/contractor  team  is  highly 
dependent  upon  success  of  both  parties.  High  quality 
practices  required  on  both  sides. 
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Which  Model  to  Use? 

SA-CMM  -  Focus  on  acquiring  a  software  system 

Acquisition  Module  for  CMMI  (new)  -  Focus  on  system 
acquisition 
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Acquisition  Module  for  CMMI 

Focuses  on  effective  acquisition  activities  and  practices 
that  are  implemented  by  first-level  acquisition  projects 
(e.g.,  System  Project  Office/Program  Manager) 

Acquisition  practices  drawn  and  summarized  from  existing 
sources  of  best  practices: 

-  Software  Acquisition  Capability  Maturity  Model  (SA-CMM) 

-  Capability  Maturity  Model  Integration  (CMMI) 

-  FAA  Integrated  Capability  Maturity  Model  (iCMM) 

-  Section  804 

Intended  to  be  used  in  conjunction  with  the  CMMI  as  an 
acquisition  “lens”  for  interpreting  the  CMMI  in  acquisition 
environments 
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Process  Areas  Included* 

Configuration  Management 

Decision  Analysis  and  Resolution 

Integrated  Project  Management 

Integrated  Teaming 

Measurement  and  Analysis 

Organizational  Environment  for  Integration 

Process  and  Product  Quality  Assurance 

Project  Monitoring  and  Control 

Project  Planning 

Requirements  Development 

Requirements  Management 

Risk  Management 

Solicitation  and  Contract  Monitoring 

Transition  to  Operations  and  Support 

Validation 

Verification 

‘Acquisition  Module  for  CMMI  expected  publish  date:  mid  Feb  04 
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Solicitation  and  Contract  Monitoring 

The  purpose  of  Solicitation  and  Contract  Monitoring  is  to 
prepare  a  solicitation  package  that  identifies  the  needs  of 
a  particular  acquisition,  to  select  a  supplier  who  is  best 
capable  of  satisfying  those  needs,  and  to  provide 
leadership  throughout  the  life  of  the  acquisition  to  ensure 
those  needs  are  met. 
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Solicitation  and  Contract  Monitoring 


The  project  is  prepared  to  conduct  the  solicitation. 

•  Designate  a  selection  official  responsible  for  making  the  selection  decision. 

•  Establish  and  maintain  a  solicitation  package  that  includes  the  needs  of  the  acquisition 
and  corresponding  proposal  evaluation  criteria. 

•  Establish  and  maintain  independently  reviewed  cost  and  schedule  estimates  for  the 
products  to  be  acquired. 

•  Validate  the  solicitation  package  with  end  users  and  potential  bidders  to  ensure  the 
approach  and  cost  and  schedule  estimates  are  realistic  and  can  reasonably  lead  to  a 
usable  product. 

Suppliers  are  selected  based  on  the  solicitation  package. 

•  Evaluate  proposals  according  to  the  documented  solicitation  plans. 

•  Use  proposal  evaluation  results  as  a  basis  to  support  selection  decisions. 

Contracts  are  issued  based  on  the  needs  of  the  acquisition  and  the  suppliers’  proposed 
approaches. 

•  Establish  and  maintain  a  mutual  understanding  of  the  contract  with  selected  suppliers  and 
end  users  based  on  the  acquisition  needs  and  the  suppliers’  proposed  approaches. 

•  Establish  and  maintain  communication  processes  and  procedures  with  suppliers  that 
emphasize  the  needs,  expectations,  and  measures  of  effectiveness  to  be  used  throughout 
the  acquisition. 

Work  is  coordinated  with  suppliers  to  ensure  the  contract  is  executed  properly. 

•  Monitor  and  analyze  selected  processes  used  by  the  supplier  based  on  the  supplier’s 
documented  processes. 

•  Evaluate  selected  supplier  work  products  based  on  documented  evaluation  criteria. 

•  Revise  the  supplier  agreement  or  relationship,  as  appropriate,  to  reflect  changes  in 
conditions. 
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Transition  to  Operations  and  Support 

The  purpose  of  Transition  to  Operations  and  Support  is  to 
provide  for  the  transition  of  the  product  to  the  end  user 
and  the  eventual  support  organization  and  to 
accommodate  lifecycle  evolution.  Eventual  disposal  of  the 
product  should  be  considered. 
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Transition  to  Operations  and  Support 


Preparation  for  transition  to  operations  and  support  is  conducted. 

•  Establish  and  maintain  a  strategy  for  transition  to  operations  and  support. 

•  Establish  and  maintain  plans  for  transitioning  acquired  products  into 
operational  use  and  support. 

•  Establish  and  maintain  training  requirements  for  operational  and  support 
personnel. 

•  Establish  and  maintain  initial  and  life-cycle  resource  requirements  for 
performing  operations  and  support. 

•  Identify  and  assign  organizational  responsibility  for  support. 

•  Establish  and  maintain  criteria  for  assigning  responsibility  for  enhancements. 

•  Establish  and  maintain  transition  criteria  for  the  acquired  products. 

Acquired  products  are  transitioned  to  operations  and  support  based  on 
transition  criteria. 

•  Evaluate  the  readiness  of  the  acquired  products  to  undergo  transition  to 
operations  and  support. 

•  Evaluate  the  readiness  of  the  operational  and  support  personnel  to  undergo 
transition  to  the  acquired  products. 

•  Analyze  the  results  of  all  transition  activities  and  identify  appropriate  action. 
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Using  the  Acquisition  Module 

Guidance  on  establishing  effective  processes  in  a  program 
office 

Informal  gap  analysis 
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Agenda 

SEI  Overview 

Capability  Maturity  Model  Integration 
Use  of  CMMI  in  Acquisition  Environments 
Conclusion 
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CMMI  and  the  Technology  Lifecyle 


Operational 

Need 


Acquisition  Module  for  CMMI 

Acquirer 


Acquisition 
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Source 

Program  Leadership 

System 

Transition 
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Developer 


Plan 
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Integrate 

Test 

Deliver 

CMMI-SE/SW/IPPD/SS 
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Contact  Information 

Brian  Gallagher 

Director,  Acquisition  Support  Program 
bq@sei.cmu.edu 

412-268-7157 
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